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FOREWORD 



The Selection & Classification Techni • ..4 of the Army 
Research Institute for the Behavioral & Z?„. Sciences (ARI) is 
concerned with developing effective procedures for the selection of 
applicants into military service and for the classification of 
accessions into Army occupational specialities. The purpose of this 
research was to examine and document the construct validity of the 
verbal subtests of the current Department of Defense military selec- 
tion and classification battery, the Armed Services Vocational 
Aptitude Battery (ASVAB) 8/9/10. 




EDGAR M. JOHNSON 
Technical Director 
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COGNITIVE PROCESSING DETERMINANTS OF ITEM DIFFICULTY 
ON THE VERBAL SUBTESTS OF THE ARMED SERVICES 
VOCATIONAL APTITUDE BATTERY 



BRIEF 



Requirement : 

To identify the cognitive processing operations, stores, and 
strategies involved in performance on the verbal subtests of the 
Armed Services Vocational Aptitude Battery ( 4SVAB) 8/9/10. 



Procedure : 

Cognitive dimensions thought to underlie performance on the 
ASVAB Word Knowledge (WO and Paragraph Comprehension (PC) items 
were oper ati onl i zed ; the verbal items on two ASVAB forms were rated 
on these variables. The relative effects of the cognitive dimen- 
sions on Rasch item difficulties were assessed for eight groups of 
FY81 Army applicants and accessions using Linear Logistic Latent 
Trait method. . 



Findings: 

Analyses suggested that cognitive processing variables were 
related to item difficulty for FY81 applicants a'nd accessions on the 
ASVAB 8/9/10 verbal subtests. The LLLT models predicted from 17% to 
30% of the variance in the item difficulty values estimated by the 
Rasch models for the WK items. The results for the PC items were 
i ncorclus iv e . 



Utilization of Findings: 

The relevance of cognitive processing variables to item dif- 
ficulty for the ASVAB 8/9/10 WK and PC subtests was demonstrated. 
Further examination of the predictive utility of these and similar 
constructs to success in Army training and military jobs is needed. 
Analyses of this type will enable assessment and documentation of the 
construct validity of ASVAB subtests. These methods can be used to 
■ievelop item sets so that specified training- and job-relevant 
processing abilities are tapped. 



J* 

vii 



COGNITIVE PROCESSING DETERMINANTS OF ITEM DIFFICULTY 

ON THE VERBAL SUBTESTS OF 
THE ARMED SERVICES VOCATIONAL APTITUDE BATTERY 



CONTENTS 

Page 

ABSTRACT T 

INTRODUCTION ■] 

METHOD 2 

The Armed Services Vocational Aptitude Battery ... 3 

The Word Knowledge Subtest 3 

The Paragraph Comprehension Subtest 3 

Part One 3 

The Development of a Model of Verbal Performance . 8 
WK and PC Item Rating Scheme 12 

Part Two 14 

RESULTS -16 

Development of the Word Knowledge Model 16 

Cognitive Component Variables 21 

Form Differences 22 

Group Differences 22 

Development of the Paragraph Comprehension Models . . 23 

DISCUSSION 28 

Relevance of Cognitive Processing Variables to Item 

Difficulty 28 

Determinations of the Parallelism of the Operational' 
Forms 28 

CONCLUSION 29 

REFERENCES 31 



9 

ERIC 



ix 

X (J 



LIST OF TABLES pa & e 
Table 

1. ASVAB 8/9/10 Subtests M 

2. Instructions and Sample Items from the Word 

Knowledge Subtest 6 

3. Instructions and Sample Items from the Paragraph 

Comprehension Subtest 7 

4. Verbal Performance Model 9 

5. Part Two Applicant Sample Sizes 15 

6. Full Model Cognitive Component Parameters: Word 

Knowledge, Form One 17 

7. Reduced Model Cognitive Component Parameters: 

Word Knowledge, Form One 18 

8. Full Model Cognitive Component Parameters: Word 

Knowledge, Form Two 19 

9. Reduced Model Cognitive Component Parameters: 

Word Knowledge, Form Two 20 

10. Full Model Cognitive Comonent Parameters: 

Paragraph Comprehension, Form One 24 

11. Reduced Model Cognitive Component Parameters: 

Paragraph Comprehension, Form One 25 

12. Full Model Cognitive Component Parameters: 

Paragraph Comprehension, Form Two 26 

13. Reduced Model Cognitive Component Parameters: 

Paragraph Comprehension, Form Two 27 



ERIC 



X 



COGNITIVE PROCESSING DETERMINANTS OF VERBAL 
ITEM DIFFICULTY ON THE iWHED SERVICES 
VOCATIONAL APTITUDE BATTERY 



ABSTRACT 



Research was designed to identify cognitive processing 
operations and strategies involved in performance on the 
verbal subtests of the Armed Services Vocational Aptitude 
Battery (ASVAB). Cognitive processes thought to underlie 
performance on the Word Knowledge and Paragraph Comprehen- 
sion subtests were identified. Items on two ASVAB forms 
were rated on these variables. The relative effects of 
cognitive processes were examined for eight groups of Army 
Applicants and Accessions using Linear Latent Logistic 
Trait methods. Analyses demonstrated the relevance of 
cognitive processing variables to item difficulty for the 
ASVAB verbal items. 



INTRODUCTION 

The purpose of this research was to identify the cogni- 
tive processing operations, stores, and strategies invoiveo 
in performance on the verbal subtests of the Armed Services 
Vocational Aptitude Battery (ASVAB) 8/9/10. Procedures 
were developed for evaluating the underlying cognitive 
ability contributions to performance on item sets. 
The methods were proposed for general use in assessing and 
documenting cognitive processing contributions to perfor- 
mance on tests and test-like tasks. The procedures that 
were developed were proposed for use in constructing 
industrial/organizational assessment instruments that tap 
specified, training- and job-relevant processing abilities. 

The specific objective of this work was to clarify 
the construct validity of verbal items on ASVAB 8/9/10. 
These results were used in subsequent analyses of the 
predictive validity of assessed ability components to 
successful performance in Army jobs. The ^research was 
conducted in two parts. 
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METHOD 

Part 1 "began with a review of recent work in the area of 
verbal information processing. The review was directed at iden- 
tifying possible sources of cognitive processing operation, store, 
and strategy differences on the verbal items of ASVAB and suggest- 
ing methodological alternatives for the examination of relevant 
data. Part 1 activities included: 

A. Conducting a review of the definition and measurement of 
cognitive processing components involved in performance on verbal 
task3 . 

B. Developing a model of verbal performance tc provide a 
conceptual framework for identifying and operationalizing process- 
ing components. 

C. Drafting theoretical and operational definitions for the 
ability components deemed relevant to performance on the Word 
Knowledge (WK) and Paragraph Comprehension (PC) items of ASVAB. 

D. Coding the Word Knowledge and Paragraph Comprehension items 
on two ASVAB R/9/10 forms for the presence or absence, complexity, 
or frequency of execution for each cognitive ability component. 

Part i methods were bared on the premise that variation in 
ifa difficulty would be a f ction of the type and level of 
required processing. It was ;o posited that processing require- 
ments could be accounted for structural features of items and 
that processing requirements could be indexed or quantified 
accordingly . 

Part 2 analyses examined the ability component contributions 
to item and person variance on the verbal items of ASVAB. Part 2 
activities included: 

A. Modeling item difficulty for the Word Knowledge and 
Paragraph Comprehension items for Fiscal Year 1981 Army Applicants 
and Accessions using linear logistic latent trait techniques; 
stimulus complexity data reflecting the nature or level of in- 
volvement for relevant cognitive processing operations and^ per- 
formance data for selected samples of examinees fed into the 
models. 

B. Cros3 validating the models across ASVAB forms and samples 
of examinees . 

A description of the operational Armed Services Vocational Ap- 
titude Battery (ASVAB) follows. 



ERIC 



The Armed IServioea Vocational Aptitude Battery . 



The Armed Services Vocational Aptitude Battery ( ASVAB) was 
introduced in January 1976 as the single Department of Defense 
selection and classification battery. Six replacement forms were 
developed and implemented October 1, 1980. The 1980 version, 
ASVAB 8/9/10, was the focus of this work. 

The ASVAB is administered to over one million applicants -For 
military service each year. The ASVAB plays two important roles 
in tne enlistment of military personnel. First, ASVaB scores are 
used in determining an applicant's eligibility for military 
service. Second, ASVAB scores are used to establish an 
individual' 3 eligibility for assignment to specific military jobs. 

The ASVAB consists of eight power and two speed tests. The 
ten subtests are listed in Table 1 . These subtests are included 
because research and experience have demonstrated that they are 
valid predictors of success in various types of military jobs. 

Table 1 also provides the number of items per subtest and the 
testing time limits. Although all ten subtests are individually 
timed, only Numerical Operations and Coding Speed are considered 
speed tests. The remaining eight subtests are power tests with 
ample time limits. 

The means and standard deviations included in Table 1 were 
jomputed for the subtest raw scores of a random sample of PY81 
Army Applicants described in Kass, Mitchell, # Graf ton, and Wing 
(1981). Estimates of subtest internal consistency reliabilities 
for the eight power subtests for each of the six forms of ASVAB 
8/9/10 were derived from Ree, Mullins, Mathews and Massey (1981). 
Parallel form reliability estimates for the two speeded subtests 
were obtained from Sims and Hiatt (1981) and Wilfong (1980). 

Tne Word Knowledge Subtest 

There are thirty-five items on the Word Knowledge subtest of 
tne operational ASVAB. The test is constructed to assess the 
examinee's understanding of the meaning of words. Surface charac- 
teristij differences associated with stem and alternatives for the 
vocabulary items include: word frequency, vocabulary level, 
number of syllables, grammatical class, stem type, etc. Instruc- 
tions and sample items from the Word Knowledge subtest appear at 
Table 2. 



Tne Paragraph Comprehension Subtest 

The ASVAB 8/9/10 Paragraph Comprehension subtest has fifteen 
items. The subtest is constructed to assess the examinee' 3 
understanding of the meaning of paragraphs. Surface chara :teris- 
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TABLE 1 
ASVAB 8/9/10 Subtests 



o 

D 
C 
<% 

a 

D 

X 

rt 

b) 

OQ 



Subtest Name 



General 
Science (OS) 



Arithmetic 
Reasoning (AR) 



Word 
Knowledge (WK) 



Paragraph 
Comprehension (PC) 



Numerical 
Operations (NO) 



Coding 
Speed (CS) 



Auto Slop 
Information (AS) 



Mathematics 
Knowledge (MK) 



Description 



Nunber of 
Itero 



Test 
Time 
(mln. ) 



20* of Amy 
Applicants for FY81 
Standard 

Mean Deviation Reliability 



A power test assessing 25 
knowledge of the physical 
and biological sciences 

Word problems 30 
emphasizing 

mathematical reasoning 
rather than 

mathematical knowledge 

A power test assessing 35 
knowledge of word 
meanings 

A power test assessing 15 
understanding of the 
meaning of paragraphs 

A speeded test of 50 
addition, subtraction, 
multiplication and 
division 

A speeded test of 84 
facility at matching 
word and number codes 

A power test assessing 25 
knowledge of auto 
mechanics, shop practices 
and tool functions 

A power test assessing 25 
knowledge of algebra, 
geometry and fractions 
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36 



11 



13 



11 



24 



14.3 C 



16.4 



23.1 



9.7 



34.2 



42.8 



14.6 



11.3 



5.2 



6.8 



7.9 



3.5 



10.5 



15.1 



5.7 



5.2 



.86 



a 



.91 



.92 



.81 



.78 t 



.85 ( 



.87 



.87 
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xnBLE 1 (continued) 



Mechanical 
Canprehension (MC) 



A power test assessing 
understanding mechanical 
principles! e.g. , 



25 



19 



13.5 



5.2 



.85 



principles dealing with 
gears, levers, pulleys 
and hydraulics 



Electronics 
Information (EI) 



A power teat assessing 
knowledge of electricity, 
electronics and radio 
principles 



20 



9 



11.1 



.82 



*Mean internal consistency reliability estimate for power subtests" (see McNemar (1969) for confutation procedure) aa 

reported in Bee, Mullens, Mathews and Massey (1981). 
b Parallel form reliability estimate as reported In 31ms and Hiatt (1981). 
cjfean parallel fonn reliability estimate as reported In Wilfor^ (1980). 

dltest statistics based on data for a 20$ random sample of PY 81 Anry applicants described In Kass. Mitchell. 
Grafton, fc Wlr« (1981). 
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Table 2 
The Word Knowledge Subtest 



This test has questions about the meaning of words. Each 
question has an underlined word. You are to decide which of 
the four possible answers most nearly means the same as the 
underlined word, then blacken the appropriate space on your 
answer sheet. 

1 . Snail most nearly means 

A. sturdy 

B. round 

C. cheap 

D. little 

2. The accountant discovered an error. 

A. searched 

B. found 

C. enlarged 

D. entered 

3. Inform most nearly means 

A. ask 

B. turn 

C. tell 

D. Ignore 

4. The wind is variable today 

A. shifting 

B. chilling 

C. steady 

D. mild 

5. Cease most nearly means 

A. start 

3. change 

0. continue 

D. stop 
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Table 3 

The Paragraph Comprehension Subtest 



This is a test of your ability to understand what you read. 
In this section you will find one or more paragraphs of 
reading material followed by incomplete statements or 
questions. You are to read the paragraph and select one of 
four lettered choices which best completes the statement or 
answers the question. When you have selected the answer, 
blacken in the correct numbered letter on your answer 
sheet. 

1. Prom a building designer' s standpoint, three things that 
make a home livable are the client, the building site, and 
the amount of money the client has to spend. 

According to the passage, to make a home livable 

A. the prospective piece of land makes little difference 

B. it can be built on any piece of land 

G. the design must fit the owner's income and site 
D. the design must fit the designer's income j. 

2. In certain areas water is so scarce that every attempt is 
made to conserve it. For instance, on one oasis in the 
Sahara Desert the amount of water necessary for each date 
palm tree has been carefully determined. 

Urn 

How much water is each tree given? 

A. no water at all 

B. exactly the amount required 

C. water only if it is healthy 

D. water on alternate days 

3. The duty of the lighthouse keeper is to keep the light 
burning no matter what happens, so that ships will be warned 
of the presence of dangerous rocks. If a shipwreck should 
occur near the lighthouse, even though he would like to aid 
in the rescue of its crew and passengers, the lighthouse 
keeper must 

A. stay at his light 

B. rush to their aid 

C. turn out the light 

D. quickly sound the siren 



ERIC 



tij differences associated with the passage, question stem, and 
aJ.ternat iveu injJ.ude: total number of words, mean grade level of 
words, standard deviation of vocabulary level, number of polysyl- 
labic words, location of response-relevant information in the 
passage, typ^ of passage (factual, fictional, first person, second 
person, etc.) average length of sentence, numVer of sentences, 
number of associated questions and type of question. The 
paragraph selections on the PC subtests have from one to four 
accompanying questions. The number of distinct paragraph selec- 
tions for the six operational ASVAB forms range, from six to nine. 
Instructions and sample items from the Paragraph Comprehension 
subtest are at Table "5 • 

Tne unit of analysis for the Part 2 computations was the 
item. The two ASVAB forms that were examined in this work each 
had nine independent PC items. The probability that the ratio of 
number of components to items would be high for the PC subtests 
was recognized. The decision to proceed with the PC analyses in 
3pite of this was based on the fact that the PC items look par- 
ticularly non-parallel across forms. The results that follow for 
the PC items were merely suggestive. 

Part One 

The Development of a Model of Verbal Performance 

The model of verbal performance that was developed for this 
work borrows heavily from the paradigms of Hunt, Carroll, 
Prederiksen, Pellegrino, Kintsch, and others. The model describes 
verbal performance in the context of text analysis. The Services 
use the verbal subtests of the ASVAB to predict general verbal 
ability; the verbal items, however, directly assess only those 
skills used in text processing. The model, therefore, charac- 
terizes verbal performance with respect to the processes which 
underlie text comprehension. The model does not define a general 
theory of cognitive processing. 

The model depicts verbal performance by five processing or 
storage structures. The first structure might be thought of as a 
perceptual processor, the second as an executive or control 
processor, the third as the locus of lexical access and semantic- 
syntactic analysis, the fourth as knowledge-based ir'orraation and 
information-free storage, and the fifth as a response 
processor. Each structure is discussed below. The structures are 
not strictly serially or hierar chic-illy ordered. The flow of 
information within the system is not necessarily sequential or 
parallel. A schematic of the mode), is at Table 4* 

The PERCEPTUAL PROCESSOR is the structure that inputs 
stimulus information to the processing system. It includes the 
set of operations that converts the physical stimulus to a form 
taut is appropriate for the task; it includes the operations that 
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Table 4 
Model of Verbal Performance 
VISUAL DISPLAY 



Perceptual Processor 

A. Visual Feature Extraction 

B. Perceptual Encoding 

C. Decoding 



Executive or 
Control Processor 




Stort Term Storage 

A. Lexical Access 

B. Semantic-Syntactic 
Analysis 



Long Term Storage 

A. Knowledge-Based 
Information 
Structures 

B. Information-free 
Functions 



Response 
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match stimuli to appropriate grapheme and phoneme 
representations. The perceptual processor is characterized by- 
three operations described by Frederiksen as: visual feature 
extraction, perceptual encoding, and decoding. Visual feature 
extraction in the operation by which different types of 
information about the stimulus display are extracted. The 
processor may select and organize information for further 
processing or it may ignore or deemphasize stimuli. Perceptual 
encoding is the operation by which information is input into the 
system; external stimuli are translated to internal codes. 
Decoding is the operation by which arbitrary physical patterns 
are recognized as representations of grapheme and phoneme 
concepts in the lexicon. These operations may be thought 'of as 
automated processes for the samples of examinees considered here. 
The processor may be thought of as a short term sensory storage 
or buffer component. 

The EXECUTIVE OR CONTROL PROCESSOR is the structure that 
control. s the flow of information in the system and has access to 
the various levels of memory storage. This structure (1) 
determines the nature of a problem, (2) has knowledge of the 
quality of one's competence with respect to a task, ("3) selects 
processes for solving a problem, (4) decides on a strategy for 
combining these processes, (5) decides how to allocate processing 
resources, (6) decides how to represent the information upon 
which processes act, (7) evaluates how well cognitive proceses 
are accomplishing subgoals in terras of overall goals, and (8) 
makes necessary shifts in processing strategies. 
This structure is analogous to Sternberg's raetacomponent and to 
the executive processor described by Snow, Whitely, and others. 

The LEXICAL ACCESS/ SEMANTIC-SYNTACTIC ANALYSIS STRUCTURE is 
a short term storage or working memory structure. This structure 
can be thought of as the workbench where information is held for 
concentrated cognitive processing. While in working memory, 
information is highly available for retrieveal. It can be held 
in short term memory while it is being operated on or transformed 
by cognitive processes. There are limitations on the amount of 
information that can reside in working memory. One's abilities 
to process such information are, therefore, limited. 

In the processing of text .naterials, a match must be made 
between letter strings input at che perceptual processing stag* 
and appropriate semantic referents. Analysis in working memory 
is directed at attaching meaning to perceptual patterns. For 
phrase and sentence units, analysis is also directed at 
organizing these meaning elements into coherent text 
representations. Lexical, semantic, and syntactic knowledge is 
cnlhed upon in the identification of words and in phrase and 
sentential analysis. 

Lexical Access is defined simply as the retrieval of 
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information about individual words from long terra memory. In 
lexical access, grapheme and phonnrae data drive the retrieval of 
semantic information. 

Semantic-syntactic analysis takes place in short terra memory; 
it is defined by the retrieval of knowledge-based structures and 
information-free functions. These structures are discussed by 
Hunt (1.978). In semantic-syntactic analysis, the knowledge-based 
and information-free long terra memory structures are accessed and, 
in the case of the inf ormation- ^ree functions, executed in short 
term memory to form a semantically coherent representation of 
prose. Information about individual words stored in long terra 
memory is retrieved and arranged to form a semantically coherent 
structure. Kintsch and van Dijk (1978) have developed a prose 
processing model which references the types of knowledge-based 
structures and information-free functions involved in semantic- 
syntactic analysis. 

The fourth structure is a long terra storage structure. This 
structure is the locus of KNOWLEDGE-BASED INFORMATION STRUCTURES 
and IN FORMAT ION- ?REE FUNCTIONS . The knowledge-based information 
structures represent semantic and syntactic knowledge. These 
structures represent the ability to deal with words and the 
concepts they represent. Tney reflect experience with and 
cognizance of the English language. The knowledge-based 
information structures are also associated with knowledge of the 
world and world events. These knowledge structures are mediated 
by verbal knowledge but represent information about the world 
ancillary to mastery 01 the English language. 

The information-free functions are the operations by which 
information structures are transformed to equivalent structures 
necessary for task performance. No semantic or syntactic 
information is associated with these strategic knowledge 
structures. A distinction is made, here, bet.,een information and 
processes. These operators are defined by ler ned , stored 
transformation rules. Examples of information-free operators are 
the processes of comparing and inferring. These operators 
perform such functions as identifying similarities and 
differences among information structures, generating missing 
bridging information to establish semantic coherence for a text, 
or sorting information structures into categories. 

The final structure is the RESPONSE OPERATOR. This is the 
structure thcougn which appropriate actions are selected and 
executed,;. The response operator is the structure by which the 
examinee makes either an observable response, such as selecting 
one response from a set of multiple alternatives or makes an internal 
response such as modifying schema in long terra memory. 
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WK arid PC Ltem Hating Scheme 



Operational definitions were developed for each of the 
operations/structures described above. These are documented in 
Mitchell (1982). The operational definitions laid the foundation 
for the following item rating scheme. For the Word Knowledge 
3ubtent, items were coded on five variables. They were: 

A. The total word count for target string, correct response, 
and incorrect response strings. This count provided an index of 
the perceptual processing load or attentional resource allocation 
load imposed by an item. It was hypothesized that increases in 
item difficulty would accompany increases in the encoding or 
attentional resource load presented by an item. 

B. Dichotomous coding for the Lexical Access/Seraant ic-Syntacti » 
Analysis variable. Context-embedded vocabulary items were' assigned 
a unit score. Non-embedded items received a score of zero. It was 
proposed that decreases in item difficulty would accompany the 
provision of contextual information for Word Knowledge items. 

C. The Kucera and Francis word frequency score for the target 
word. This value indexed the probability that stored information 
was available for the target word. It was hypothesized that 
decreases in item difficulty would accompany increases in word 
frequency scores for the target word. 

D. The Kucera and Francis word frequency score for the correct 
response. This value indexed the probability that stored semantic 
information was available for the correct response string. It was 
hypothesized that decreases in item difficulty would accompany 
increases in word frequency for the correct response. 

E. Count of the high frequency-correct response/ low frequency- 
incorrect response pairs for each item. The Kucera and Francis 
word frequency scores were used for the correct response and 
incorrect response strings to generate a count of the high-correct 
response/low-foil response pairs. This value indexed the relative 
complexity of judgements of semantic identity for correct 
response/ target word and incorrect response/ target word pairs. It 
was hypothesized that decreases in item difficulty would accompany 
increases in the number of high frequency-correct response/low 

f requer jy-incorrect response pairs. 

The ve -iables for the PC items were: 

P. The total word count for the paragraph, correct response, 
and 'incorrect response strings. This count indexed the perceptual 
processing or attentional resource allocation load presented by an 
item. It was hypothesized that increases in item difficulty would 
accompany increases in the encoding or attentional resource load 
presented by an item. 
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0. The number -jf text-baaed inferences necessary to maintain 
semantic ooherence as defined by Kintsch and van Dijk (1978) for 
the target paragraphs. The construction of text-based inferences, 
a controlled, mechanistic information-free function, calls upon 
knowledge-based information structures. A description of the 
rating system appears in Turner and Greene (1978). It was 
proposed that increases in item difficulty would accompany 
increases in the number of text-based inferences necessary, to 
maintain semantic coherence. 

H. A dichotomous coding for the response inference variable. 
Items for which the construction of an inference was necessary to 
linK the correct response to information presented in the 
paragraph selection were unit coded. Coding followed Turner and 
Greene (1978). Again, inference construction is a controlled, 
mechanistic information-free function which draws upon knowledge- 
based information structures. It was hypothesized that item 
difficulty would be positively related to the construction of 
response inferences. 

1. The number of propositions per paragraph selection divided 
by the number of words in the selection. Coding, again, followed 
Turner and Greene (1978). This value indexed the complexity of 
proposi tional structure for a paragraph selection. It was 
hypothesized that increases in item difficulty would accompany 
increases i*" number of propositions suggested by each 
paragraph selection.. 

J. The number of arguments per paragraph selection divided by 
the number of words in the selection. Argument coding agreed with 
Turner and Greene (1978). Like variable four, this value indexed 
complexity by processing for a paragraph selection. It was 
hypothesized that increases in item difficulty would accompany 
increases in the number of different arguments presented by each 
paragraph selection. 

K. The mean of the log transformed Kucera and Francis word 
frequency indices for the arguments of each selection. The value 
indexed the probability that stored semantic information was 
available for the arguments of a paragraph selection. It was 
hypothesized that decreases in item difficulty would accompany 
increases in mean word frequency scores for paragraph selections. 

Obtaining ratings for the 5 variables on the 70 Word 
Knowledge items and the 6 variables on the 18 Paragraph Comprehen- 
sion items took approximately 80 riter nours. 24 hours were 
needed to obtain a second set of ratings on variables G-J for the 
18 Paragraph Comprehension items. Percentage agreement rate3 for 
codings on Paragraph Comprehension variables G, H, I, and J were 
hit, U%, and 92<£, respectively. 
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Part Two 

The relative effects of the cognitive processing variables on 
the item difficulty parameters for the verbal items of the A3VAB 
were estimated using the linear logistic latent trait model 
proposed by Fisher (1973). The model is described by Whitely and 
Schneider (19*1) and Thissen (1982). 

The model is a one-parameter latent trait model. Item diffi- 
culties or location parameters are estimated using: (1 ) item 
response data for samples of examinees, and (2) an item rating ma- 
trix reflecting structural characteristics of the items. In these 
analyses, the item rating matrix was the item-by-cognitive component 
array described in Part One. The matrix defines a set of linear con- 
straints on the estimation of the item location parameters. 

The one-parameter latent trait model estimates item difficulty 
in the following way: 

« er i;L*n_I_°il 

P(x lj' 1) "T+ exp( t 3 - 

where ^Cj -the ability level for person X> 

e± -the difficulty for item _i- 

The linear logistic latent trait model consists of a set of linear 
constraints on item difficulty paramet e rs , ■ such that, 



where is a vector of difficulties, f;^ is the 

item-by-cognitive component rating matrix, 0^ is a vector of 
component parameters, and Cl is a normalization constant. The 
component parameters reflect the relative effects of the cognitive 
•component variables on the item location parameters. 
The complete linear logistic model is given: 



When the £ im matrix is an identity matrix and x is set to 0, the 
linear logistic model is equivalent to the one-parameter latent 
trait, or Rasch model. In this case, each item defines a separate 
; item difficulty is the only item parameter. 

Part Two analyses focused on eight samples of examinees 
tested on ASVAH 8/9/10. Six of the samples were FYB1 appliaants; 
two samples were FY81 Army accessions. Samples one and two were 
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Army applicants who we^e randomly selected without regard to 
whether they were actually admitted to the Service. Sample one 
was composed of 2998 applicants administered form one of the 
A3VAB. Sample two had 2925 records; these applicants took form 
two. 

In addition to these two samples, subsamples of applicants 
defined by level of verbal ability were examined for each form. 
Prom che two samples above, the twenty percent with the lowest 
Word Knowledge and Paragraph Comprehension summed score were 
selected and the twenty percent with the highest Word Knowledge and 
Paragraph Comprehension summed score were selected. The N's for 
these low verbal and high verbal ability samples for each form are 
at Table 5. 



Table 5 

Part Two Applicant Sample Sizes x 

A S V A B 

Verbal Ability Groups 

Form 1 Form 2 

Heterogeneous Ability 2998 2925 

Low Ability 585 610 

High Ability 604 609 

These four groups were subsamples of applicant samples one and two. 

Two samples of FY81 Army accessions were also selected. The 
samples of accessions were students enrolled in three high-density 
Array job training programs. The occupational specialties were: 

i • q 5B Military Police 

q 4B Food Service Specialist 

"5. 76C % 76P Equipment Records and Parts Specialist and 

Material Control and Accounting Specialist 

Accession sample one had 400 examinees; sample two had 358 records. 

Hasch item difficulties and linear logistic latent trait item 
locations were estimated for applicant and accession groups using 
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the marginal maximum likelihood procedures outlined by Thissen 
( 1 9S2 ) . 

For the linearly constrained latent trait model, component 
parameters were also calculated for each form and sample. The 
component estimates from the latter analyses were examined, and 
cognitive component variables with sizable contributions to tne 
estimation of item difficulty values were retained in a reanalysi 
of the data. For each group, this third analysis was computed 
usin* the linear logistic latent trait algorithms. A reduced 
model was defined by the third analyses. The statistical sig- 
nificance of the individual cognitive component parameters was nc 
assessed; estimation of standard errors for the component 
parameters is presently computationally intractable. 

The adequacy of the cognitive component variables in 
predicting difficulty parameters for the Word Knowledge and 
Paragraph Comprehension items was assessed by correlating the 
Rasch item difficulties with the linear logistic latent trait 
difficulties and the reduced linear logistic latent item 
locations. The Rasch item difficulties, computed with one 
parameter per item, functioned as criterion scores for each 
sample. The two sets of linearly constrained logistic latent 
trait item locations functioned as predictor scores. The two 
simple correlations indexed the strength of the relation between 
the unconstrained and contrained item difficulties estimated in 
each applicant and accession group. 



RESULTS 

Development of the Word Knowledge Models 

The full and reduced models for the Word Knowledge subtests 
are at Tables 6-9. The correlations between the Rasch item 
difficulties and the five-variable linearly-constrained item 
difficulty values for the Word Knowledge subtests ranged from .45 
to .55. There were essentially no differences in the correlations 
ajross forms or subjects. 

Reduced linear logistic latent trait models were developed 
for each Word Knowledge form and sample. Three sets of values 
werP considered in the selection of cognitive component variables 
for the reduced models. They were: (1) the cognitive component 
parameters estimated by the linear logistic latent trait algo- 
rithms, (?) the intercorrelations between items ratings on the 
fivp Word Knowledge variables, and (3) the simple correlations 
between the item ratings and the Rasch or unconstrained item 
difficulty values. 
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Table 6 



Full Model Cognitive Component Parameters (df=34) 
Word Knowledge, Form One 
FY8l Applicants FY81 Low Ability FY81 Hi Ability FY81 Accessions 



1. Total Word Count -.761 

2. Lexical Access/Semantic- .481 
Syntactic Analysis 

3. Target Word -.829 
Frequency 

4. Correct Response -.076 
Word Frequency 

5. High Frequency-Correct .102 
Response/Low Frequency- 
Incorrect Response Pairs 

Normalization Constant -.329 

SLmple r's, Rasch/Linear Logistic .48 
Latent Trait Item Difficulties 



Applicants 
-1.967 
1.133 

-1.338 
-1.353 
.491 



3.923 
.15 



Applicants 
.111 
.420 

-1.367 

-.079 

.166 



-2.782 
.55 



-.756 
.635 

-1.231 

.040 

.114 

-1.078 
.49 
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Table 7 

Reduced Model Cognitive Component Parameters (df=34) 
Word Knowledge, Form Che 



1. Total Word Count .253 

2. Lexica" Access/Semantic- 
Syntactic Analysis 

3. Target Word -.784 
Frequency 

4. Correct Response 
Word Frequency 

£ 5. High Frequency-Correct 
Response/Low Frequency- 
Incorrect Response Pairs 

Normalization Constant -.865 

Simple r's, Rasch/Linear Logistic .44 
Latent Trait Item Difficulties 



FY81 Applicants FY81 Low Ability FY81 Hi Ability 

Applicants 
.251 



-1.048 
-.830 



2.72 
.41 



Applicants 
1.197 
-1.647 



-3.891 
.49 



FY81 Accessions 
.707 



-1.127 



-1.945 
.45 
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Table 8 

Pull Model Cognitive Component Parameters (df=34) 
Word Knowledge, Form Two 
PY81 Applicants FY81 Low Ability FY81 Hi Ability FY81 Accessions 



1. Total Word Count .415 

2. Lexical Access/Semantic- -.602 
Syntactic Analysis 

3. Target Word -.024 
Frequency 

4. Correct Response -.445 
Word Frequency 

5. High Frequency-Correct -.002 
Response/Low Frequency- 
Incorrect Response Pairs 

Normalization Constant -.685 

Simple r's, Rasch/Linear Logistic .51 
Latent Trait Item Difficulties 



Applicants 

.434 
-1.674 

.162 

-1.788 

.224 



3.047 
.46 



Applicants 

1.049 

-.720 

-.152 

-.746 

.190 

-4.508 
.53 



1.152 
-.916 

-.039 

-.569 

-.050 

-1.951 
.55 



32 



33 



o 

ERIC 



O 



1. Tbtal Word Count 

2. Lexical Access/Semantic- -.'172 
Syntactic Analysis 

3. Target Word 
Frequency 

4. Correct Response -.455 
Word Frequency 

5. High Frequency-Correct 
Response/Low Frequency- 
Incorrect Response Pairs 

Normalization Constant -7366" 

Simple r's, Rasch/Linear Logistic .50 
Latent Trait Item Difficulties 



Table 9 

Reduced Model Cognitive Component Paiameters (df=34) 

Word Knowledge, Form Two 

FY8l Applicants FY8l Low Ability FY8l Hi Ability 

Applicants 



Applicants 
-1.681 

-1.689 



4.195 
.45 



-.477 



-.831 



-4.687 
.51 



FY81 Accessions 
-552 

-.623 



-1.142 
.54 
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Cognitive Component Variables 



The cognitive component parameters for both the Total Word 
Count and Lexical Access/Semantic-Syntactic Analysis variables 
were sizable for all examinee groups and both forms. These 
variables were entered singly in the reduced models because item 
ratings for the two variables were highly correlated. The word 
count for an item was positively related to whether the target 
word was content embedded or non-embedded. The Total Word Count 
variable was selected for the he tereogeneous and low verbal 
ability applicants and for the accessions taking form rne . The 
Lex ical Access/ Semant ic-Sy n tactic Analys i3 variable was included 
for the high anility applicants taking form one and for all 
examinees administered form two. 

The sign of the cognit ive component paramete r for the Total 
Word jount variable in the form one reduced models was positive. 
The cognitive component parameters for the Lexical Access/ 
Semantic-Syntactic Analysis variable for form two analyses were 
negative. The direction of effect was consistent with 
hypothesized relations for both variables. Item difficulty was 
positively related to the total number of words presented by an 
item. Item difficulty decreased with the provision of contextual 
information for the Word Knowledge items. 

There was evidence of interacting effects for the Total Word 
Count and Lexical Access/Semantic Syntactic Analysis variables 
when they were entered simultaneously in the five-variable models. 
A possible interpretation of the interaction is that item dif- 
ficulty does not increase with heavy encoding requirements when 
the additional lexical units provide context. Because the decre- 
ments in the item difficulty correlations from the five-variable 
linear logistic latent trait models to the reduced models were 
minimal when the variables were entered singly, these two vari- 
ables were not combined in the final models. 

The cognitive component parameters for the Target Word 
Frequency variable for form one and the Correct Response Word 
Frequency variable for form two were sizable. The Correct 
Response Word Frequency component parameter for the low verbal 
ability group taking form one was also large. The Target Word 
Frequency variable was entered in the reduced models for all 
groups taking form one. The Correct Response Word Frequency 
variable was included for the low verbal ability group taking form 
one and for all examinees taking form two. 

Like the Total Word Count and Lexical Access/Semantic-Syntac- 
tic Analysis variables, item ratings on these two variables were 
highly correlated. Difficult target words were accompanied by 
difficult defining words. The direction of effect was consistent 
with hypothesized relations for these two variables in the reduced 
m^ielr*. Words with low frequency counts were positively related to 
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item difficulty both for the target word and the correct alterna- 
tive. 



The cognitive component parameters for the High Frequency 
Correjt Response/Low Frequency Incorrect Response variable were 
small for all groups. This variable was operationali zed in such a 
way that its values were highly related to the values assigned on 
the Correjt Response Word Frequency variable. 

The parameter estimates for the cognitive component variables 
could not be evaluated for significance. In addition, the prob- 
abilities would be meaningless . : .n light of the model-building 
procedures used. Only the relative effects of the cognitive 
processing variables included in the models could be examined. 

The correlations between the reduced model item difficulty 
estimates and the Rasch item difficulty values ranged from .41 and 
.54. Tne correlations were quite similar to the correlations for 
the five-variable models. The models explained from 17^ to 30^ of 
the variance in the item difficulty values estimated by the Rasch 
models. 



Form Differences 

The reduced models for the two forms of the Word Knowledge 
subtest were judged to be conceptually similar. Both sets of 
models contained a variable which indexed the amount of informa- 
tion presented in the item and both included a variable which 
indexed the difficulty of the target or response words. 

The Total Word Count variable is a measure of the amount of 
information that must be encoded, analyzed, and retained by the 
reader. The Lexical Access/Semantic-Syntactic Analysis variable 
indexes the provision of contextual information for the target 
word. For the Target Word Frequency and Correct Response Word 
Frequency variables, item ratings reflect the probability that 
examinees have encountered and comprehended information about 
individual lexical units. For both forms item ratings for the 
Total Word Count and Lexical Access/Semantic Syntactic Analysis 
variables and the Target Word Frequency and Correct Response Word 
Frequency variables were highly correlated. 



Group Differences 

Although differences in estimated Rasch and linear logistic 
latent trait item difficulty values were observed for examinees 
differentiated by level of verbal ability, no marked differences 
between the cognitive component parameters were noted for groups 
of examinees on the two forms. The composition of the models and 
relative magnitude of the cognitive component parameters were 
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jomparable for hetereogeneous, low, and high verbal ability FY81 
applicants and PYB1 accessions. 



Development of the Paragraph Comprehension Models 

The results of analyses on the Paragraph Comprehension 
subtests were difficult to interpret. Analyses were performed on 
nine items for each subtest and relations were hypothesized for 
six cognitive processing variables. The ratios of the number of 
parameters to the number of items for both the full and reduced 
models were such that meaningful interpretation was prohibited. 
The results below are presented merely for completeness. The full 
and reduced models appear at Tables 10-13* 

Cognitive component variables were selected in the same way 
for the reduced Paragraph Comprhension models as they were for the 
reduced Word Knowledge models. The cognitive component parameters 
for the # Proposit ions/# Words, # Arguments/^ Words, and Mean 
Argument Frequency variables for form one were high for the 
hetereogeneous ability level applicants, the high ability ap- 
plicants, and the accessions. For the low ability group, the 
absolute values of the Word Count, Response Inference, # Proposi- 
tions/^ Words, and Mean Argument Frequency component parameters 
were high. 

The correlations between the six-variable model item dif- 
ficulties and the Rasch item difficulty estimates ranged from .81 
to .93 for Form One. Correlations of this magnitude were presup- 
posed by the fact that the number of parameters was only slightly 
less than the number of items. For the reduced models, the 
correlations ranged from .67 io .84* There were no discernable 
differences between groups. 

For Form Two the correlations between the linear logistic 
latent trait difficulty estimates and the Rasch item difficulties 
ranged from .86 to .^9. The cognitive component parameters for 
the Response Inference, # ?roposi tions/# Words, and # Arguments/^ 
Words variables were high for the hetereogeneous applicant group, 
the low ability group, and the accession group. The Response 
Inference variable did not have a sizable weight for the high 
ability group. 



The correlations between the 
difficulty values for the reduced 
ficulties ranged from .81 to .96. 
f>6^ to 92^ of the variance in the 
the Ra^ch models. There appeared 



linear logistic latent trait 
models and the Rasch dif- 

These models represented from 
item difficulties estimated by 
to be no group differences . 
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Table 10 



Full Model Cognitive Component Parameters (df=8) 
Paragraph Comprehension, Form One 
FY81 Applicants FY81 Low Ability FY8l Hi Ability FY8l Accessions 



1. Total Word Count 
'2. # Text-Based Inferences 

3. Response Inference 

4. # Propositions/^ Words 

5. ft Argunents/# Words 

6. Mean Argument Frequency 


-.078 
.174 
.462 
-4.400 
-4.652 
-2.339 


Applicants 
-1.698 
.153 
1.935 
-4.246 
.410 
-1.772 


Applicants 
4.597 
.647 
-.427 
-12.850 
-16.702 
-6.480 


.810 
.269 
.660 
-6.855 
-6.197 
-2.636 


Normalization Constant 


2.792 


7.218 


-2.074 


1.922 


Simple r's, Rasch/Linear Logistic 
Latent Trait Item Difficulties 


.93 


.84 


.89 


.81 
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1. 

2. 



6. 



Total Word Count 
# Text-Based Inferences 
Response Inference 
ft Propositions//? Words 
ti Arguments//? Words 
Mean Argument Frequency 



Table li 

Reduced Model Cognitive Component Parameters (df=8) 
Paragraph Comprehension, Form One 
FY81 Applicants FY81 Low Ability FY81 Hi Ability 



Normalization Constant 

Simple r's, Rasch/Llnear Logistic 
Intent Trait Item Difficulties 



FY81 Accessions 





Applicants 


Applicants 






-1.878 








1.875 






-3.393 


-3.245 


-4.037 


-4.415 


-6.265 




-11.242 


-8.792 


-3.529 


-2.156 


-6.860 


-4.359 


3.291 


7.523 


2.418 


4.047 


.84 


.83 


.67 


.69 
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Table 12 



Pull Model Cognitive Component Parameters (df=8) 
Paragraph Comprehension, Form Two 



FY8l Applicants 



1. Total Word Count 

2. # Text-Based Inferences 

3. Response Inference 

H. ft Propositior.V# Words 

5. ft Arguments/^ Words 

6. Mean Argument Frequency 

Normalization Constant 

Simple r's, Rasch/Linear Logistic 
Latent Trait Item Difficulties 



FY81 Low Ability 
Applicants 



-.494 


-9.274 


6.319 


-.033 


.286 


-.243 


.718 


3.582 


.288 


3.240 


-33.083 


30.152 


-1.250 


6.813 


-4.880 


-.572 


.936 


-1.923 


-.654 


30.702 


-25.807 


.99 


.86 


.95 



FY81 Hi Ability FY81 Accessions 
its 

1.239 
-.130 
1.166 
11.507 
-6.072 
-1.039 

-6.398 
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Table 13 



1. 
2. 
3. 

4. 

5. 
6. 



"total Word Count 
U Ttext-Based Inferences 
Response Inference 
0 Propositions/^ Words 
# Argiments/# Words 
Mean Argument Frequency 



Reduced Model Cognitive Component Parameters (df=8) 

Paragraph Comprehension, Form Two 

FY81 Applicants FY81 Low Ability PY81 HI Ability 

Applicants Applicants 

1.729 



Normalization Constant 

Simple r's, Rasch/Linear Logistic 
Latent Trait Item Difficulties 



.739 
1.7^0 

-.139 



-1.480 



• 9o 



2.940 
-17.633 
4.707 



7.558 



.81 



18.215 
-743 



-13.800 



.87 



FY81 Accessions 



1.199 
5.249 
-3.273 



-2.805 
.96 



DISCUSSION 



Relevance of Cognitive Processing Variables to Item Difficulty 

Part 2 analyses suggested that cognitive processing 
variables were related to item difficulty for FY81 Army 
applicants and accessions on the verbal subtests of the 
ASVAB. The reduced linear logistic latent trait models 
predicted from 17% to 30* of the variance in the item 
difficulty values estimated by the Rasch models for the 
Word Knowledge items. The reduced models were predictive 
of the Rasch-computed item difficulties for the Paragraph 
Comprehension items. 

Additional information about the relations between 
cognitive processing requirements and item difficulty might 
be gleaned in future research using item sets specifically 
developed to represent a varied range of proficiency on the 
individual processing variables. More sensitive assessment 
of individual differences on these variables and of their 
relevance to item difficulty might be attained with 
instruments constructed with special attention to assessing 
a varied range of ability levels on the processing 
variables. Evaluation, here, was possible only on the 
cognitive constructs represented on the operational ASVAB 
and only in terms of the component variance built into the 
test . 

Further information about the relations between 
cognitive processing variables and item difficulty might 
also be gained by fitting multidimensional item response 
models to these data. The present analyses fit 
unidimensional item response models to data which were 
multidimensional. Both theoretical and empirical 
considerations attest to the multidimensionality of the 
data . 



Determinations of the Parrallism of the Operational Forms 

There was no evidence to suggest that the two 
operational forms of the ASVAB under investigation were not 
parallel. There were essentially no differences in item 
ratings for the five Word Knowledge and six Paragraph 
Comprehension variables across the two forms. The 
cognitive component examinations of the two forms did not 
point clearly to differences between forms. 
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CONCLUSION 



The relations of cognitive processing variables to item 
difficulty for the ASVAB 8/9/10 Word Knowledge and 
Paragraph Comprehension subtests were demonstrated. 
Assessments of the predictive utility of these and similar 
constructs to success in Army training and military jobs 
are needed. f 

It is proposed that analyses of this type allow for the 
assessment and documentation of the contributions of 
cognitive processing operations, stores, and strategies to 
performance on industrial/organizational assessment 
ins'-ruments. These techniques can inform evaluations of 
the predictiveness and relevance of aptitude and ability 
measures. These methods can be used to develop item sets 
so that specified, training- and .iob-relevant cognitive 
processing abilities are tapped. 
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